Task-Oriented Vision with Multiple Bayes Nets

We present the basic framework of a task-oriented computer vision system, called TEA, that uses Bayes nets and a maximum expected utility decision rule. Knowledge about the scene and about the nature of the specific task given to the system are represented in Bayes nets . We introduce a new kind of Bayes net, called an expected area net, that models both geometric relations between objects and the areas in the scene where objects are expected to be located. The decision of what areas of the scene to run a vision module on can be made using this relational knowledge. The decision of what vision modules to run is made using a value/cost utility measure, where value is based on mutual information measured between nodes in the Bayes net that correspond to actions and to the goal of the task, and cost is proportional to the fraction (using the expected area net) of the image that a vision module processes. We present a method for combining several Bayes nets that represent different types of scene and task knowledge into a single composite network. Composite nets support more complex visual tasks, and enable the calculation of action utilities relative to the information requirements of a specific task. TEA models camera movements and distinguishes between vision modules that operate either on foveal or on peripheral image data. Experimental results are presented from the TEA-O system, our initial implementation of the general TEA framework . This material is based upon work supported by the National Science Foundation under Grants numbered IRI-8920771 and IRI-8903582. The Government has certain rights in this material.